Recommender systems are well known in the art. In one example, such systems can make recommendations for movie titles to a subscriber. In other instances they can provide suggestions for book purchases, or even television program viewing. Such algorithms are commonplace in a number of Internet commerce environments, including at Amazon, CDNOW, and Netflix to name a few, as well as programming guide systems such as TiVO. While the details of such algorithms are often proprietary, the latter typically use a number of parameters for determining a user's movie “tastes” so to speak, including demographics, prior movie rentals, prior movie ratings, user navigation statistics, comparison with other users, etc.
Recommender systems are often implemented as collaborative filtering (CF) algorithms. Such algorithms purportedly are “content” neutral, in the sense that they provide recommendations to a user for an item based on his/her similarity to another user (or users), and not with regard to the characteristics of the item itself. CF algorithms nonetheless may not be entirely “neutral, and may include subtle unintended (or even intended) bias in their recommendations. In some cases they may not recommend items that are “new” because CF systems tend to lag in their learning capabilities.
From the perspective of a subscriber or a content provider, determining the existence and extent of bias in a particular recommender system may be important. For example, a movie studio, a book publisher, a television program source (i.e., different types of content provider) may want to determine if a particular content service provider is accurately presenting recommendations to the right demographics group.
A recent article by Kushmerick titled “Robustness Analyses of Instance-Based Collaborative Recommendation”—13th European Conference of Machine Learning, 2002, incorporated by reference herein—makes mention of the fact that recommender systems can be potentially “attacked” by outsiders to artificially inflate or degrade ratings of items. This problem is treated as one of “noise” which can affect the reliability and reputation of recommender systems. A similar discussion is presented by Kushmerick et al. in another article entitled “Collaborative Recommendation: A Robustness Analysis” —ACM Transactions on Internet Technology, Special Issue of Machine Learning for the Internet —(publication date unknown), which disclosure is also incorporated by reference herein. Thus the problem of “noise” added to recommender system datasets is just beginning to be appreciated.
Notably, however, Kushmerick fails to consider the possibility of an internal “bias” which is intentionally introduced by the recommender system operator, or how to detect/measure the same. Such bias may be designed and built in by the recommender system operator based on a desire to alter—i.e., boost or reduce the marketability of certain items in exchange for some incentive from a third party. Since such bias is introduced by the operator, it is extremely challenging to detect from the outside. Nonetheless, the identification and measurement of such bias is clearly useful to outside parties to help gain an understanding of the relative fairness, reliability, reputation, etc. of recommender systems.
Furthermore, a content provider may want to test the adequacy and suitability of an inventory management and/or shipping system used by a particular service provider, to ensure that their stock of items is being adequately managed. From the perspective of a content provider, it is important to improve the efficiency of distributors who are effectively managing consumer demand for items by the content provider. One important parameter, for example, may be the issue of how quickly a recommender system for a particular vendor is able to assimilate and give recommendations on new items. The lack of data for new items is a known limitation of recommender systems, and yet the prior art does not describe any mechanism for comparing the performance of recommender systems in this respect.
In addition, the prior art does not consider how to determine whether a recommender system is complying with a particular preference policy which might be specified for recommendations. Such mechanism can afford a purchaser of such preference an opportunity to determine the performance of an online operator in achieving/satisfying a particular marketing/advertising criterion.
Finally, the prior art does not indicate how the effects of advertising can be correlated with recommender system behavior, or even how recommender system recommendations can be mined and exploited to improve online advertising campaigns. Accordingly, there is a present need for systems and methods for achieving such functions.